Stochastic Weight Averaging Revisited

Averaging neural network weights sampled by a backbone stochastic gradient descent (SGD) is simple-yet-effective approach to assist the SGD in finding better optima, terms of generalization. From statistical perspective, weight-averaging contributes variance reduction. Recently, well-established (SWA) method was proposed, which featured application cyclical or high-constant (CHC) learning-rate schedule for generating weight samples weight-averaging. Then, new insight on introduced, stated that average assisted discovering wider optima and resulted We conducted extensive experimental studies concerning SWA, involving 12 modern deep model architectures open-source image, graph, text datasets as benchmarks. disentangled contributions operation CHC showing SWA still contributed reduction, exploring parameter space more widely than SGD, could be under-fitted due lack training budget. then presented an algorithm termed periodic (PSWA) comprised series operations exploit such wide structures explored schedule, we empirically demonstrated PSWA outperformed its remarkably.